Testing the Maximum Entropy Principle for Information Retrieval

A probabilistic information retrieval method using the Maxfrom Kantor, 1994), or (b) useful for information retrieval, imum Entropy Principle (MEP) was proposed by Cooper even if it is not true. In the present article, we examine these and Huizinga (1982). Several refinements of the MEP for questions using the TREC5 (Harman, 1996, in press) biinformation retrieval have been proposed by Kantor and corpus. In particular, we have concentrated on the 50 routing Lee (1986, 1991), but the MEP has not been evaluated in topics, and on documents which have either been judged as any large database. This article examines the MEP retrieval method using the TREC5 database. The MEP is evaluated to relevance, or are contained in 0.1%, 1%, and 10% random by several tests and compared with a ‘‘naive ordering samples of the TREC5 documents. method’’ and ‘‘lexicographic ordering method.’’ The MEP There are several ways in which the MEP might be does not provide any startling improvement, and it works found to apply. The Strong MEP asserts that the actual reasonably well only in the case of a small number of keys distribution of relevant and non-relevant documents and a relatively small collection. across the collection corresponds, as a probability distri-